Skip to content

Conversation

MenSeb
Copy link
Owner

@MenSeb MenSeb commented Oct 28, 2023

PULL REQUEST

Description

Please describe the reason related to the changes:

Breaking Change

Does your changes introduce a breaking change?

  • no
  • yes

Information (issue, links, logs, etc.)

Please list any other useful information:

@MenSeb MenSeb changed the base branch from main to html-tracing October 28, 2023 04:11
@MenSeb MenSeb changed the title build(html-tracing) implement user agents utilities build(html-tracing) implement user agents and session utilities Oct 28, 2023
@MenSeb MenSeb merged commit 398800e into html-tracing Oct 28, 2023
@MenSeb MenSeb deleted the user-agents branch October 28, 2023 22:49
MenSeb added a commit that referenced this pull request Dec 11, 2023
* build(html-tracing): initiate html tracing package

* build(html-tracing): setup Flask app and implement proxies logic (#2)

* build(dependencies): install dependencies for Flask

* build(flask): setup minimal server with Flask, templates and statics files

* build(dependencies): install dependencies to fetch and parse an url

* ci(ruff): config pydocstyle with numpy

* build(pandas): install pandas and update python version for numpy

* ci(gitignore): update gitignore with temp and datas folders

* build(proxies): implement proxy utilities

* ci(coverage): remove coverage threshold

* style(lint): fix missing new lines

* build(html-tracing) implement user agents and session utilities (#3)

* docs(proxies): update docstring description of proxies

* docs(useragents): create user agents class and update docstring description

* build(useragents): implementing user agents helpers

* docs(poxies): update docstring documentation

* build(user-agents): add helpers to extract a list of user agents

* docs(proxies): move docstring descrption to proxies

* build(session): create session helper to request using proxies and agents

* refactor(agents): renaming user agents file

* build(session): update session messages log

* build(utilities): create refresh helper to fetch and format

* build(utilities): rename helpers functions names

* docs(utilities): update docstring description

* build(html-tracing): implement methods to clone a website (#4)

* build(agents): remove logger

* build(session): update session with internal function

* build(clone): implement functions to clone a website

* build(proxies): enable reuse of dataframe in query

* build(logger): implement logging utilities

* build(logger): update logger logic with tracing

* build(logger): update logging with new line config

* build(logger): update logger with internal logging

* build(logger): use logger in utilities methods to visualise progression

* build(clone): skip fetch if file exists

* build(session): add delay between request

* docs(session): update docstring with delay time

* build(session): update request with delay for access and read

* refactor(clone): update clone logic

* build(packages): enable imports from different packages

* refactor(project): update project structure

* build(versions): update poetry and python versions

* build(session): update session logic and docs

* build(logger): update logger logic and docs

* build(benchmark): update logic with logger

* build(ruff): update ignore rules

* build(test): test logic using helpers

* build(scraping): update scraping with refresh and save logic

* build(spider): initiate web spider logic

* build(tracing): test web spider from server and client

* build(prettier): create prettier config file

* build(poetry): update poetry version and install dependencies

* build(scraping): update logic and add check logic for available proxies

* build(aiohttp): install dependencies

* build(server): test server logic

* build(yaml): update yaml linting

* build(server): test server logic
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant